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PREFACE , 



The booklets in this resourte guide were? designed to provide 
information about concepts, techniques and strategies that ca 
assist minority* xearounities in developing and evaluating drdg 
abuse 'prevention programs to meet the specific needs bf their 
neighborhoods * barrios, reservatipns, and towns. 

These Jfe^oklets are not "how .to" publications'. Every cc 
when viewed in terms of its needs, preferences, financial 
resources, and most importantly, inner strengths, is /unique 
There *re no standard, blueprints to address Jthe needi of -eg 
ethnic conrauoiity. The members of the coqraunity mustf work 
closely together to find solutions to common problems* We|hope 
that these booklets will provide some guidance to beginning 
process of coranunit)^ involvement in drug abuse prevention/ 

s. f 
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*6ie words ''minority; 1 and ,r ethnics of color"! will be used inter- 
* changpably in this booklet, Many people fefel that the teim 

M In^Jlority , / refers* to a' status Of powerlessrfess within the system, 

and thus prefer "ethnics of color." 1 
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THE PURPOS E OF THIS GUIDE - 

— — % 

.Jhis booklet addresses some issues relating to the evaluation 
of drug prevention programs for minorities. It is designed to 
encourage a thoughtful approach to/program evaluation that takes 
into consideration the cultural orientation of the target popula- • 
*tion in: * ^ < 

• planning for evaluation; , 

• determining the role of the evaluator; ^ 

• selecting an evaluation methodology; ^ 

• selecting or designing data collection iijstruments ; 
% selecting and training staff for data collection; 

• collecting and analyzing the data; ^ , 
\% reporting the findings; and * "~ 

• using. evaluation results for program improvement. 

This booklet has been developed primarily for prevention pro- 
gram planners and administrators who are not particularly know- 
ledgeable about program evaluation. It does not present the defrn- 
> itive model for evaluating mtfrtiailtural drug abuse prevention - 
none exists. It should be afesefui tool, however, t;o assist pro- 
grams in working with intamalSur external evaluators. Although « 
not designed specifically for them, evaluators will be interested^ 
in reviewing this and other booklets in the series for a discussion 
of multicultural ccdicerns in evaluaticm. ¥ 



While a highly technical discussion on implementing evaluations 
I is beyond the scope of this booklet, extensive references on re- \ 
4 search and evaluation among minorities and a bibliography are pro- 
, vided. Other resources which should prove helpful are NIDA's N 
recently published "Prevention Evaluation Guidelines" and its 
National Prevention Resource Network JNPERJjF, One West iVilson Street, 
Room 434, Madison, Wisconsin 53702) < 



- Human service* programs, especially in the prevention field, 
are increasingly being required to conduct evaluations , as part 
of a cc-ntinuing movement toward tlje use of evaluation as a tool 

. for program improvement. The demand of policy makers and funding J> 
sources for drug abuse prevention programs to prove they "work" is 
particularly high. Good evaluation designs in {his area, there- 
fore, will continue to be sought. Such evaluations must include 

4 t£e same cultural- considerations required for planning and- imple- . 
menting programs that have been discussed in the crther bboklets 
in this series. * ^ 



Sensitivity to cultural issues is particularly cnicial in the 
evaluation of prevention programs. This is true becauska val id 
evaluation is basically a mechanism for communicating program 
.effects to decision-makers and persons interested in adopting the i 
program. The evaluation must accurately identify, describe and y 
ggfefcplain program effe€ts on the target population',, the social 
^Bystem within wftic& the program is operating, and" the community. 
This information must also* be correctly translated to the inter- 
ested parties in a useful form. *- 

Cultural groups exist in unique social^and psychological 
contexts which shape>attitudes, values, behavior, view of self , 
and the world. It is essential that these contexts be integrated 
into the development and implementation of evaluations of multi- 
cultural drug abuse prevention programs. 

Research apong minorities, however, is still in its infancy, 
and program evaluators are just beginning to become sensitive to * 
cultural issues. Many programs targeted tn minorities continue 
to be evaluated with research approaches more appropriate to 
Afiglo-American populations. Some research has been done on Blacks, 
Hispanics, American Indians/Alaskan Natives, and Asian American/ 
Pacific Islanders.* But there is a severe problem in assembling 
and disseminating what little information there 'is to evaluators 
of minority programs. The inability to make program evaluation 
culturally relevant comes at a time when minority leaders are 
dema/iding that govejument sponsored programs he responsive to 
their communities • needs. A cred/ble, culturally relevant "eval- 
uation can help a drug abuse prevention program bridge the gap 
between meeting the funding agency* s demands for accountability 
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and being responsive to the ethnic community by adjusting and ' 
improving services to better meet' their needs. <f 



^PROGRAM EVALUATION: WHAT AND WHY 

Numerous terms have been coined by evaluators to communipate 
the purpose, jpract ices, an4 features of their own version of 
evaluation. The labels used, however, are far less important than 
the quality of the information produced. Simply put, program 
evaluation is a management tool which provides information* for 
measuring and improving the effectiveness of program performance. ( 
It has two major functions: 4ecision-making and accountability. 
To some extent, evaluation involves measurement in that data are 
Systematically collected and analyzed. But it also involves judg- 
ment because the data can provide only "the basis for decision- • 
making. Decisions based upon sound evaluation procedures, however, 
are more credible than those based on unstructured impressions 
because" iocumeqted evidence can be provided to substantiate them, 
this does nqt mean that statistics are essential.' As pointed out 
in later sections of this booklet, qualitative ^information describes 
progress more usefully for many' programs. 

A successful evaluation is not necessarily one which declares 
that all of the program's goals have been met, but rather is one 
which tells what an£ how the program is doing and why. To be/ 
useful in^decision-making, an evaluation should describe and. 
explain wfiat^the drug abuse prevention program is doing. It should 
provide 'facts which objectively describe the activities of interest 
and explain why .things are as they are. „-In doing so, the evalua- 
tion should not 'focus attention on explanations based on the 
attitudes and behaviors ef the participants to the exclusion of 
the socio-political environment — that is, institutional, social, 
and cultural arrangements that impinge on them. 

With these ingrecKentfs , program plans can be made, procedures 
can be Changed, activities cap. be added or deleted/ and support 
can be justified and obtairiKi from bath the funding agency and the 
coniTimityy' Thus, evaluation findings can be 'used' to improve a 
program 1 f public image. Such improvement and decisions can be 
made during the on-going operation of the program and near or at 

^the, end of the program year. Even quite negative findings can be 

'useful for determining corrective action. 

Among the practical reasons for evaluation are: 

'*# C to identify what kinds of action work most 
effectively to meet program goals; 

/ • to avoid conducting activities which are unrelated 
to goals and objectives; 




* « 

• to analy&e program strengths and Weaknesses; 

• to identify and describe the inputs/processes/ ^' 
% , * outcomes, stated and jmstated; 

, • to convincingly attribute- the /impact to program,, I — 

•activities; [ * . ^ *J ^ # 

% • to provide data for- reports to various groups such 
as minority community representatives, advisory 
groups, and funding agencies; ^ 

• m v^to meet requirements for program evaluation ' , 
C " mandated by a state or federal agency as a 

condition of funding; and " ' 

* * • * 

a , • to provide, information for policy decisions, program r 

0 ' " planning, and program imprpvement. V 

% , : • \- <. , - 

Other roles for evaluation, in addition tto the above, can^be 
^ t6 advance the state of the art in multicultural d*ug abuse pre- 
Y vention concepts and practices and to develop methodologies to 

improve- evaluations of multicultural drug abuse prevention programs . 

'Evaluation can involve intricate statistical designs and 
. procedures.' In cases where program evaluation is being carried 
ou£ on a relatively small Scale, however, overly sophisticated 
techniques are not necess&iy. ^The complexity of the evaluation 
-should be dipectly related to the cojflplexity of *the program and 
s y ' the nature of the management decisions to be'made. 

Throughout this. booklet, evaluation is discussed from a po>r=~ 
/ tive perspettive.^ However, not ^11 evaluation efforts are for the 
"good of mankind, or science." There are some pitfalls and abuses 
t that exist and pl^nner^ and eva^uators , particularly those that ,are 
( conmmity based, need to be aware of the most comnon ones. 



Those tftat are the most obvious and occur most frequently 

/» 4 

1/ ,T Eye-wash" --Attempting to justify a weak or ineffective 
^ program by deliberately selecting only those portions that f 

. • ^ !, look good". , • , \~ 

2. "White-wash" -- Avoiding any objective appraisals-. ^ 

, 3. ~* Postponement"' — Using the study as a delaying'tactic. , 
• * t 

4, "Posture" Trying to look professional" for the public. 

v 5. . ,t Subfnarine ,, -% Using evaluation to destroy a program. 

x ■ t ^ 

• .... -J -4- • - . * 
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'Just as a substantive knowledge of the purpose and 'eventual 
use of the evaluation help determine the 'techniques and process " 
used, this same knowledge helps prevent us from falling into the 
evaluation ! £raps" lis tgd. a" . r ' ' 



iLEVELS'OF EVALUATiaN 



.^The Prevention -Branch of the National Institute, on BfOg 
Abuse (NIDA) uses a program assessment model that specifies 
three levels of evaluation; process, outcome, and impact. The 
levels may be considered useful in working with various program 
molalities W target populations. The relationship among level 
of evaluation, target group, and modalities is represented in 
the drug abuse prevention matrix in Exhibit 1. , 



Process Level 

* J Process evaluation assesses the methods used to implement the 
'program and*is defined by NIDA as follows: 

& . ' 

• Process information reflects the inputs* that go into a* 
program, the patterns in which these inputs interact,^ 

' . -and'the transactions that take place within the pro- 
gram. 'Information such as participant and staff t? 
characteristics, physical plant characteristics, arid 
» financial resources, as well as the theory^on which 
the program operates, needs assessment, policy devel- 
opment, and program* design activities are all examples t 
of program inputs. .Information derived from the socio- 
political environment is also considered to be impor- 
tant evaluative information because of its potential 

* contribution to subsequent evaluation and its use as 
a basis for recordkeeping systems. Others sessments 
on the process* level may include a description of - 
services rendered, the decision-making structure, 
patterns of interaction among participants and staff, 
and so pn (French, et. al., 1979, p. 10). 

'^y^ professionals feel that the way in which pro- 

cess data is collected should reflect a sensitivity to the culture. 
This important fact may determine whether or not accurate "true 
informatidh is being collected. Use of written "objective 
measurements may not be useful, for instance, in a commmity 
that has an established oral tradition. Anita Arkeketa who is 
a specialist in training American .Indian program staff suggests 
that "...oral assessment (of training) proved to be less intimi- 
dating and tinie consuming and allowed participants to share 
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^DRUfi fflUSE PRFVFNTIflN MATRIX 
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.EVALUATION 
TYPE 



/ . 



IMPACT 



OUTCOME 



TARGET 



PEERS 

I . 

SCHOOLS 

OTHER SOCIAL 
INSTITUTIONS 




M 0 JW L I T Y 



PROCESS /INFO". /EDUC. SALTERN/ INT , 
INDIVIDUAL 
FAMI-LY 




learning in a leisurely manner... "* Her„ observations, in working 
with Indian programs ih # the Southwest suggest that participants 1 
level of involvement and quality of participation is enhanced 
when oral techniques are used. New learning in this area 'can, be 
^ facilitated when program developers and evaluators search for and 
create innovative approaches for, collecting such data,' 

Process data provfcief decision -making information durihg the 
life of the program as well as information for interpreting out- 
comes. It focuses on the questions: Is the program proceeding-** 
as planned? What problems exist in meeting the needs of the 
comnunity? / Are program strategies working? Are they culturally 
relevant? What are the attitudes and opinions of staff and 
others regarding the prQgram? 

Outcome Level 

*» 

Outcome Evaluation assesses tlje accomplishments of the program. 
The following definition is used, by NIDA: 

r , * 4 *. 

Data gathered during this phase of program evaluation 
typically are addressed to specific program objectives 
concerned with change in participant behavior, atti- 
tudes, values,- or knowledge. The' major objectives' 
in all prevention program modalities concern the f 
reduction of inappropriate drug and alcohol use. ^ 
At the same time, different prevention programs 
have unique objectives relating to the particular ' 
theories underlying them. These include such ♦ 
diver^objectives as ijnprovement of self -concept 
. and responsibility, reduction of alienation, in- 
crease in achievemerit%>tivation, and improvement 
in a broad range of variables relating to school 
performance. And this list is far from exhaus- 
tive (French, et. al., 1979, pp 10-11). 



Outcomes are the accomplishments of the program not only at 
its end, but also during its life* If data are collected during 
the needs assessment on individuals who will be directly affected 
by the program these data can be used as a baseline for comparison 
with the outcome data. For example, the needs assessment data may 

^include measures of attitudes, opinions, or behaviors of teachers, 
youth workers, and potential program participants that can be used 

' to assess changes in these individuals after the program^ is 
implemented. In order for this to be effective, any data that are 
to be compared should be collected in the same manner, using com- 
parable methodologies and data collection instruments. Needs- 
assessment data have an even broader use as baseline data for the 
itopact evaluation which_is discussed in the next section. 

^Training of trainers course modification for an American Indian 
population. Anita Arkeketa, Southwest Regional Support Center* 
1980. 
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s "Some questions to be addressed by the evaluation of outcomes 
include: What prevention assistance is availably to the culture 
that was not available befdre the program began? What, problems 
have been resolved? Were program objectives met? Were there any 
measurable changes in the behavior of the target population 'that ' \ 
can be attributed to the program? What were the outcomes of the 
program that were not planned or expected? Certainly all of these 
will have different manifestations based S)n the culture .in which 
the program resides. Tests for measuring self-concept*, for example, 
should be validated to use with certain cultures. When no such 
tests are available, # program developers should seek technical \ 
assistance in modifying existing measurements to match such needs. 
Hie NIDA NPERN project has compiled ijists of evaluators whc-have ^ ^ 
skills in these areas. NPERN can make 'specialists available to 
programs interested in exploring the--* use of new measurement . 
techniques. • * H ; , 

Some of the ethnic social science organizations (see Booklet 4 : 
Resources) have published bibliographies and lists -of persons 
wfto have knowledge and have done research in the development of 
instruments. ' * , 

<. * * *@ * 

This phase of evaluation can 'assist our program concepts and T 
contribute new and valuable information about prevention from a 
cultural- perspecti^. . " * 

Impact « Level • 

Impact evaluation looks at longer-term, community-wide . 
effects of the program. NIDA defiAes* it as follows: '- a 

/ * 
Information gathered in this phase relate^ to longer- 
term, generalized results of program operations. The 
manner in which impact data aresrelayed is*a function 
m ' of the comnunity needs and problems which gave rise to 
the prevention program in the f^rst place. -That is why 
such broad 4 issues as changes 'in incidence and prevalence 
" in d^ug abuse and in contnunity competence to deal with 
these problems are frequently addressed in*impact eval- 
uation. SucA changes impinge directly on inputs to the 
program (French, et. al., 197S, y>. 11). • 

Impact refers to the genera^ ^effectiveness of the program in 
meeting the needs of the comnudity as defined by the needs assess- 
ment data. Therefore, the sa^fcn4s'pf data are often collected 
in the needs assessment and the impact* evaluation. Approaches 
mentioned in Booklet 2 ; Needs Assessment of this series are: 

• • > K^INTORMA\T APPRi^^* Soliciting views, opinions, 
and fact A s from key informants in $he comnunity. 

*The term M key informant" is used in evaluation to refer to indivi- . 
duals who are in a key position to asses? a program. Key informants 
may be community leaders, parents, teachers, social workers, etc. 




• SURVEY RESEARCH APPROACH: ' Surveying selected m . 
populations in the coSamity (survey research). 

x • RATEg-UNDER-TREATMENT APPROACH : Analyzing institu- 
4 ¥ tional records to determine rates -under- treatment 
(socio-demographic, characteristics of health and 
mental health clients). , 

« SOCIAL INDICATOR APPROACH : Inferring v and estimating 
service delivery needs* f ran descriptive statistics 
found in public records and reports. • (Caution is 
advised here. While some indicators are valid for 
the Anglo- American culture or even for some other * 
ethnic cultural group, they may not be *for your 
target population}. 
\ 

Exhibit 2 graphically displays these>at>proaches. 

By comparing the needs assessment, data and the impact data, 
changes in the target population afid the comnunity can be assessed* 
As previously mentioned, however, the methodologies for these two 
data collection efforts must generate comparable data. This 
requires some long range planning on the part of the funding agency, 
the program staff, and the evaluator. 

Assessing long-range impact that is, proving that the. drug 
jgfiise prevention program, really 'Vorks" -- in a scientifically 
valid way is exceedingly difficult. As pointed out earlier, 
this is what policy makers and funding sources most offen want * 
from an evaluatidn. An imgact evaluation, howevey, requires 
long-term sophisticated methodologies to allow for many ^ 
factors that can inf luence* or obscure the findings. This kind of 
evaluation is beyond the capabilities and resources of most indivi- 
dual programs and must be left to the larger programs or a national 
effort. It, is not, therefore, discussed at any length in this 
booklet. 

* ■ « 

Envi ronmental Consideration? JCfr 

Another important consideration in the evaluation of a program 
is its cultural and socio-politipal environment. In this regard, 
some things to be considered are: \ 

the dominant culture of the community; i*e', Who has power 
over the system and resources. 

the cultural orientation of the program; e.g., rural Black, 
migrant Puerto Rican, etc 



the service delivery system within which the program 
functions; e.g., education, mentaj. health, health, etc. 
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• the political climate of the conrnmity; i.e., to what 
extent are members" of the culture allowed access to 
resources and power. 

• the political climate q£ the service delivery system; 
e.g., does the program have the sanction of the^ 
conmunity. ^ 

The socio-political environment can generate both positive 
and negative influences on the program. Awareness of these 
environmental influences is important to any program evalua- 
tion. See Booklet 2: jfeeds Assessment for a fuller discussion. 

The Role of the Evalua(qr „ x 

- - r~ * , 

An outside or third-party evaluator is in most instances, 
preferable .to an internal evaluator because he/she is external 
to and independent of the system being evaluated and, therefore, 
should be npre objective. However, a^third-party evaluator is 
likely to be more costly thaj? an internal evaluator. 

* • 

The use of an external evaluator does not indicate that 
internal efforts are invalid. Program personnel in minority 
programs need to know that their evaluation efforts are meaning- 
ful and "truth" is not defined soley from the outside. They 
should be encouraged to learn more about measuring program in- J 
fluences and the elements and methodologies used in the evalua- 
tion process, so they can work effectively with a/i outside 
evaluator and/or produce. internal product^ that meet their needs. 

Tne "objectiv><y^ of the evaluator is<a major concern of 
minority program operators. Because prevention programs reach 
directly to the core .oJLethnic cultures and operate in the 
realm of values, family traditions, and social patterns, it is 
.essential that any third-party evaluator take special care in 
understanding the life-style and cultural value systems of 
conmunity members being studied. Bryon Kunisawa has developed 
a technique for, assisting persons in assessing their awareness t 
and sensitivit/ to other cultures. His Multicultural Survival 
Inventory, which is contained in Appendix A, can be used to 
assist third-party ^valuators in conducting a ( self- assessment 
priof to working with a given corounity. ^ 

This inventory is suggested only as a potential tool. 
Because it implies an object ivf -assessment derived from a 
Western European base, some persons may choose not to use it, 
or may use only a^pa'rt of it. ^ 

Regardless of ethnic background, an evaluator for a minority 
drug prevention program should have the following characteristics: 

• knowledge of and experience in using a variety of 
evaluation strategics; 
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. • innovativeiiQSs? in 'developing research designs for special 
populations; ) • 

t a track reco|6 Sfor performing credible evaluations; \ 

• a sensitivity to the ethnic and cultural context of the 
program; 

• the ability, to incorporate ethnic considerations into 
the evaluation design and instruments so that the social 
distance between the*evaluators and the respondents is 

. minimized; » 

• willingness ta train and work with indigenous data 
collectors; and • ^ 

• the ability to comnunicate and work well with the program 
staff. 

r^rm^IL ar ^ SeVeral °P? i ? ls conc erning Aether and under what ' , 
circumstances to use a third-party evaluator or an internal * 
evaluator. Exhibit 3 illustrates six options for conducting process 
and outcome evaluations using third-party and internal evaluators. 

* Ik vary by ^ md de * ree of objectivity. In Option 1, 
- the third-party evaluator conducts all activities related to the 

process, and outcome evaluations including plan design, and the 
collection and analysis of the data. This option has an added 
advantage if the evaluator has the expertise to provide technical 
assistance to the project as part of the process evaluation*, Third- 
aParty process evaluation with technical assistance is an atJpfbach 
often used for programs in other fields.- In Option 2, the flftrd- 
party evaluator designs the plans and analyzes the data for tShe 
process and outcome evaluations, but trains and supervises thfe 
internal evaluator or project staff to collect the data. W process 
evaluation is done by the internal evaluator in Option 3, while the 
outcome evaluation is done by, the third-party evaldator. Although 
this is a frequently used approach, it can result in considerable^ 
duplication of effort if the two evaluations tare not carefully 
coordinated. Option 4 represents a method trfat provides an 
acceptable amount of objectivity at a relatively low cost. In 

* this approach \the process evaluation is done by the internal 
evaluator. The third-party evaluator designs the plan and analyzes 
the data for tl# outcome evaluation, but trains and supervises the 
evaluajor-or project staff y> collect the outcome evaluation^. 
Since the internal evaluator is doing much of the evaluation, and 
aata collection, there is an opportunity to coordinate the two * 
evaluations and still maintain the objectiyity required for the 
outcome evaluation. In Option S, both the process and outcome 
evaluations are done internally. Only^n internal outcome eval- 
uation is done in Option 61 Both of the latter options will 
encounter problems concerning, the credibility of evidence of pro- 
gram ipict. For a small project with a limited budget, Option 4 
has tip most to offer. Qption 2 is jnore preferable but is also 
more costly. * r % 
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EXHIBIT , 3 

OPTIONS FOR USING THIRD- PARTY AND/OR INTERNAL EVALUATO^s" 



Option Option Option ^Option Option Option 

1 2 3 4 __5 — 



Third-Party Evkluator 



Designs process plan X X 

Collects process (Jata X v 

Analyzes process data XX \ 

Designs outcome plan X X X X * % 

Collect outcome data X j X % 

Analyzes, outcome data X X X 

Internal Evaluator : ( 



Designs process plan X -X X 

Collects process data X X X X 
Analyzes process data X X I X — 

Designs outcome plai> _ XX 

Collects outcome data X XXX 
Analyzes outcome data X ~ X 



The evaluator' should work closely with the program director 
and staff to ensure that ethnic and cultural aspects germane to 
the program are appropriately incorporated into the evaluation 
design. The staff should have input into the evaluation plan, 
the selection of a methodology, the selection or desigri of eval- 
uation instruments, the data collection, and the- analysis. 

* i 

PLANNING FOR EVALUATION * 



Ideally, planning for evaluation begins when the program 
itself is' being planned. If a needs assessment process such as 
that outlined in Booklet 2 of this Series is followed, the pro- 
gram will have identified needs and defined the <?iltural and socio- 
political environment in which they exj*t. Program goals will 
have been determined after which objectives could be specified. 
Those objectives would be both clear and measurable as well as 
realistic in order to serve as the basis of the evaluation. 
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EXHIBIT 4 ^ \ 


WRITING CuTTURALLY RELEVANT PREVENTION PROGRAM OBJECTIVES . 


CCfWON CONSIDERATIONS 


CULTURAL CONSIDERATIONS , 


Objectives should: " 

1. Monti fy results or conditions to be achieved 
rather than activities to be performed. 

2. Be limited in time so as to provide nule- 
* stones of achievement. 

^» he stated in terms of what ts fco be done 
rather than in terms of what is to be 
avoided. 

, 4. Be designed to co\er a single end result. 

Read 1 1 vindicate a- data baseline, * 

Be written tn quantifiable terms that*nre 
measurable in terms of established 
standard^. 1 

Indicate the minimum level of achievement or 
"standard" tha-t is acceptable. 

8. Fit within the framewWk of the overall goals < 
and policies of the program. \ 4 

9. Be realistic and attainable, ^ _ 

2J 

consistent with resources «av£ilable 
gj^j^>ind anticipated. 


L. How^s achievement viewed and valued" ^ > 

2. \re the project timelines realistic 
and attainable? 

3. Is there a history of having planned activities 
imposed on the commuriity through negatively 

'stated sanctions (religious, educational, etc.)? 

4. I.s thqre universal agreement among the cultural 
group on the individual end result desired? 

5. Is there a realistic data base line? How- 
can one ^j^pjevp loped 0 

6. Are there cultural standards? How can 
established standards he more inclusive" 

Are the standards achievable as well as relevant 
to the culture?| Has there been an acknowledge- 
ment of^those stancjards by formal/informal 
lender** 0 

8. Are the overall *goa Is and policies in line 
with the culture? 

9. Realistic within cultural constraints? Attainable 
^ with respect to cultural needs and desires? 

10. 4Vhat resources are availsb&e yithja the culturfe? 
How can these resources be accessed? 



1 




U is critical that the evaluator, whether internal or 
exterrerfcj^jnd the key people reiatewi to a project understand and 
agree upon the definition of success with respect to each objective. 
The objectives, therefore, should be reviewed to ensure that they 
are realistic and feasible. There is a natural and understandable 
tendency to let enthusiasm foj programs overcome coigpon sense, 
and this leads to impossible objectives that may wellfcportend 
♦ the program 1 s failure even before starts. A drop-in center 
with a hotline, for example, cannot realistically expect users 
of that line to become self- actualized and more achievement _ 
oriented. This feasibility review should also cover any economic, 
political, or social constraints imposed by the dominant culture 
on the specific sub-culture that would impede achievement of the 
objectives. /The importance of establishing objectives in the 
psycho -social realm that are reflective of those standards and 
attributes va^ufed by the ethnic culture rather than those of the 
dominant culture cannot be emphasized strongly enough. Exhibit* 4 
summarizes considerations in writing culturally relevant preven- 
tion program objectives. 1 / 

If objectives are realistic and culturally relevant, thty J 
should be examined again to ensure that they are also measurable. 
Such objectives specify t«he target population, the behavior they 
will exhibit Cbehavior is much easier to assess than attitudes, 
values, self-iragej , the numbers or n/^rcentages. Time and condi- 
tion are two -other elements of a measurable objective. Time 
merely "means the date by which the results as defined ar§ to be 
attained. The objective should also specify under what condi- 
tions the defined* results will be obtained , as well as conditions 
t under which the results would not be expected. Anticipating such 
'conditions in advance eliminates the "appearance of looking for 
excuses after the evaluation is completed. 

. In developing measurable objectives, rigid or absolute values- 
of project achievement are frequently less realistic than a range ' 
7 of' such values. If the criterion for success, for example, is 
~)that 50 per cent of the target population will participate in 
alternative activities, should 49 per cent be considered failure ^ 
to achieve the objective? It would be wiser to establish a mini- ' 
mum level (say S5 per cent) as well as the maximum anticipated 
(50 per cent). 

Qnce ,the e valuator and program staff have analyzed the objec- 
tives and agreed upon the criteria and standards for success, 
they, are ready to develop the balance of the evaluation plan. 
Other' components in designing an evaluation are^ 

• Select the appropriate methodology; 

• Determine availability of inforjnation; 

• Determine how data will be collected, when, 
and by whop; 4 



-15- 

• 23 



ERIC 



• Determine how data will be analyzed; 
* • Determine how findings will be used/presented; and 



• Review and keep the evaluation plan current. 

In general, a good evaluation plain is one which provides 
valid and reliable ipfoimation for making the necessary deci- 
sions with the least disruption to the program and at the* least 
.cost. It should fit well and be compatable with the overall 
♦program design. The plan should alSo provide for adequate 
resources and support In terms of personnel, materials, time, 
and program cooperation. Providing inadequate support is a 
threat to the validity and reliability of the evaluation data 
and can be a wabte qf money. On the other hand,* overly designed 
evaluation plans that require the collection of superfluous data 
should be avoided as they are often too cftstly and too disrup- 
tive to the program. * 



i^e steps in program evaluation are detailed, with appro- 
priate exhibits, in the next section. 

m . ' „ * 

A helpful aid in all facets of planning and program opera 
tion is the Gantt chart. The^Encyclopedia of Management 
defines the Gantt chart as follows: 

"The Gantt Chart is a visuaJL management control device 
developed during World Wpr I by Henry L. Gantt, one of 
the pipneers in scientific management. It is a linear 
calendar on which future time is spreacC horizontally 
and work to be done is indicated vertical!/."' 



In any activity, the only constant is t^me, and therefore 
the scale of the Gantt chart is time — future time 
"* the calendar spread horizontally across a sheet. Any 
suitable divisions and subdivisions of time can be used 
months, weeks, days -or hours." 

In addition, a sample Gantt chart of preliminary evaluation 
activities can be found in Appendix B. 
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ISSUES IN THE EVALUATION OF MINORITY PROGRAMS 

The evaluation of minority programs raises several r 

issues in the following* areas: 

■ ' + 

" • identifying the most relevant variables for the 

Valuation design; 

• maintaining reliability and validity; - K + 
• • gaining entxry to the minority comnunity; * 

m • * gaining the s.upport of the communitly; ^ 

• decreasing the social distance between program ' % 
« evaJ^uators and respondents; an4 ♦ 41 

m • ^increa^ing the understanding of the evaluation results- 

Quantitative and qualitative approaches to the issues are 
discussed under each of these headings below. 

I. Identifying the most relevant variables fqp&he evaluation 
f design. ' 

The most relevant variables for the evaluation of a V 
minority drug abuse prevention program can be identifi<ed by 
using ethnographic and other investigative strategies in an 
evaluability assessment* 4 

The purpose of this assessment is to plan for a more 
useful evaluation. During this I^riod, program characteristics 
are identified, the. completeness of records is assessed, the 
availability of respondents is. investigated, constraints that 
wi4l affect the evaluation are identified, and alternative 
methodologies are reviewed (Rutman, 1980). 

- Ethnographic methods are used to discover and explore 
different facets of the program within its cultural context. \ 
Preliminary hypotheses are formulated, reformulated, and confirmed 
in an iterative fashion. Behaviors and interactions are 
observed in naturalistic settings. (Rudes, Blair, et al,*, 
1980). . . 

The information from the evaluability assessment is then 
used to develop the research design. Researchers who conducted 
the assessment are u3*d throughout the evaluation to increase 
the reliability and validity of the findings. 
«i 
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Mdiktaining reliability and validity 



The ultimate measure of expected impact of a prevention 
program is that drug use was either prevented or ""reduced in 
the target population over a long term as a result of the 
•program. Since long term reduction of drug use in the target 
population is usually beyond the scope of most program 
evaluations, interest often focuies on more immediate program 
outcomes ^such as'ch3nges in attitudes toward drugs or improved 
self -concepts. \ However, changes or improvements in human, 
behavior may be attributable to many factgrs other than 
participation in a drug prevention program. The natural 
maturation of the individual is a stropg influence in behavior^ 
improvement §mong children, adolescent s,/*and young' adults. 
Maturation can even be a factor in behavior change among 
adults. Because of the emphasis on the psycho-social realm 
in drug abuse prevention, most , programs include objectives ' , 
.that address personality characteristics, attitudes and opinions, 
social or interpersonal behaviors. These are legitimate as 
long as they are defined in the context of the Individuals ' . 



assumption that behavior within norms of another culture isf 
itself deviant because it'differs from the dominant culture. 

A relatively controlled experimental design is required 
to reliably attribute effects on the target population to ' 
the program. Program administratorsj^ed this kind of credible 
evidence of program effects in ordei^) make decisions regarding 
policy and' funding. Thus, some form of quantitative 
methodology ^is needeld (Rudes, et al., 1980). 

A quasi-Gocperimental design ,with comparison groups-""^ ^ 



should be used to measure program effects on the target 
population if at all possible, because the use of comparison 
groups will provide the most credit^e evidence of the program V 
effects. In fact, the use of two comparison groups rather* 
than one would provide stronger evidence of program effects 
for a minority program because the effects of cultural in- 
fluences could oe as3essed, Using this approach, the first 
group woul4 include program participants of whatever ethnic 
background tie program serves* (e.gr, Puerto Rican program 
participants). Group two wouid be comprised of persons from 
the same ethnic background who did not participate in the 
program (e.g., Puerto Rican non-participants). Persons from 
the dominant culture who did not participate in the program 
would ^e included in group three (e>g>, Anglo-Ameriean non- 
participants) . # It is assumed that the groups would bejnatched 
on t such salient characteristics as .age, sex., social diss, 
academic achievement, etc. a*s much as possible. 

Using a Puerto Rican program, as an illustration, fr the 
following comparisons would be possible: * 





6 Puerto Rican program' participants with Puerto Rican 
non-participants; 



" • . ° ' 9 

0 

• - . ' r - 

• Puerto Ricta program participants with Anglo- 
American non-participants; and 

*•« 

• Puerto Rican non-participants with Agio-American 

, ^ non-participants. '* * 

These comparisons would supply information on how Puerto Ricans 
are doing in relation to each other, and how they are doing in 
relation to Anglo-Americans. In this manner, Puerto Rican, 
ethnicity as a factor in the behavior can be evaluated to some 
degree and biases in measurement instruments may be more 
readily identified." 

If it is only possible to have one comparison group, the 
group should be of the same ethnic background as the program 
participants. * In such an instance, group three would not be 
used. 

T^ere are several methods for selecting comparison groups 
which are addressed in the evaluation literature and, therefore, 
will not be discussed'in this document. Suffice it to say 
that whatever method is used, the criteria and methods for 
selection should be well documented and become part of the 
evaluation, recordkeeping system. The record should indicate 
how the groups were selected, on what characteristics they 
we?e matched, find the statistical and analytical procedures 
used to account for variations within and between groups. The 
• program evaluator should provide expertise in this area. 

Quantitative methodgfthat do. not use comparison groups are 
also discussed in the evaluation literature', but these methods 
do. not provide the credible evidence of program effects that 
use of comparison groups-yields ,£1^ / 



The validity and understanding *0f the process and outcome 
evaluation results Can be ^enriched .by using a quantitative 
methodology in conjunction with information ob tailed using 
qualitative methods. For example, in-depth, relatively un- 
structured interviews with key ^informants can provide, a better 
understanding of the process evaluation findings. Interviews 
with or case studies of a sample of program participants 
can be a valuable aid in explaining program outcomes. Informal 
observation of; or participating in, conimmity or advisory 
ponmittee meetings js also useful. „ 

Z+ Gaining* entry td the minority oonrnunity ' . « 

It is very important to become familiar with the social 
organization and mores of .thfe minority community within 
which, a prevention program operates in order to conduct a needs 
assessment or 'evaluate the effectiveness of a program. There 
may be a need to conduct interviews with minority Igaders , 
, parents, or youth in the community. For example, Tsukashima 
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(197/, pp* 133-143) used a two stage research design. In the 
first stage, he used ethnographic methods to get a feel for 
the comnunity and to identify significant variables. In the 
second stage, he conducted interviews with a randan sample 
of community leaders. Strategies for gaining entry will t vary 
considerably dep&ding on -the ethnic composition and charact- 
eristics of the comnunity, the persons in the community one 

wishes to contact, and the objectives of the research. 

' * • 

Contacts cari ofiten be v formed in the community through 
afquadntenances such as field workers, students, parent^ 1 
groups, etc. If the same person's name is menti{Jfed by several 
people, that person .may be worth talking tp. 

Previous- research indicates that it is important to 
establish ap^ijignfcjiy that satisfies the community and allows 
for maximun J^novation for the researcher. It is also 
important to make the purpose of the research clear. Weiss 
(1977, pp. 120-132) found that the low key role of the naive 
but friendly strange^ was effective with traditional, older 
Chinese but was suspect among young activists. Many minority 
activists find it difficult to tolerate uncommitted strangers. 
Weiss thinks that non- involvement may be viewed as a political 
stance by some groups.. ^ 

It cannot be assumed that' an ethnic comnunity is 
homogeneous , The community may include a mixture of ethnic 
groups and within each ethnic group there will be differences 
.by social class and the degree of assimilation of its members 
into the dominant culture, as well as generational differences. 
There also may be several conflicting factions in the cpmnunity. 
The researcher's role will be interpreted differently by the 
different groups in' the community and these' interpretations 
may be deeply imbedded in the culture. 

4. Gaining the support of the oommnity 

Closely related to gaining entry to the community is the 
problem of gaining community support for the evaluation. The . 
importance of maintaining community support should not be under- 
estimated. Community representatives have the means to influence 
funding agencies regarding the program and any research affiliated 
with it. The purpose of the evaluation is not only to provide . 
information to the funding agency hit also to iniprove services 
to the comnunity and to provide information to the comnunity 
regarding the Effectiveness of the program. 

%s previously mentioned, the purpose of the evaluation 
should be made clear to the ethnic community from tne beginning. 
An effort should be made to find out what type of information , 
is of interest to members of the community and in what form 
they prefer it. Then the evaluation plan should be designed 
to provide^ for the information needs of all users including 
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members of the coinmunity. 

Input from community members should be maintained through- 
out the evaluation process in order to increase the validity, 
understanding, and acceptability of the evaluation. 

< 

5. Decreasing the social distance Vetween program evaluators 
and respondents 

Infective communication with the minority target population 
a program serves is as crucial to a valid evaluation as it is 
to the successful implementation ©f the program. The evaluator 
must be able to tap the cultural perspective that influences 
individual behavior and translate it into the evaluation 
design, thus minimizing the social distance between the , 
evaluator and tie respondents. This means that the choice of 
language used in the evaluation should reflect the style and 
variety used by the^target population. The type of information 
being collected and the manner in wliich it is collected should 
reflect the cultural perspective of the target population. 

This is not an easy task because there will^ differences 
in language and perspective within the same ethnic group and 
between different ethnic groups in the same community. Each * ' 
program will have its own unique characteristics ' that must be 
understood before it can be adequately evaluated. m 

Ethnography provides a useful approach for maximizing 
effective communication between evaluators and persons involved 
in the evaluation. This methodology provides a mechanism for 
understanding the ethnic value structure manifested in verbal * 
and physical communication which is. crucial for. eliciting valid 
evaluation information. Comrnunication between evaluators 
and respondents, between the program staff ^nd respondents, between 
the evaluators and the program staff, and other -significant 
cannunication networks must be« considered. The use of language 
and gestures are part Qf the COTinunication system. 

Research indicates that there are ethnic differences in J t 
the use of social and temporal space. For example', Asians 
typically do not maintain eye contact with authority figures, 
but rather look down a little. Also, Asians are not as touching 
oriented as sane other cultural groups. The acceptable physical 
space between two people is greater for Asians than it is for 
Americans and Hispanics. Sane Native Americans have culturally 
acceptable pauses or periods of silence in conversations, 
particularly between the t,ime a question is asked and an answer 
is given. These periods of; silence can easily be misinterpreted. 
There has been considerdble "research on other types of non-verbal 
cannunication (including the study of gesture, facial expressions, 
stance, and tone of voice).' Some non-verbal communication is 
culturally based' while sane is based in the individual's person- 
ality. Thus, it is worth an investment of time for the . 
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evaluator to become familiar with the cultural" context of 

the program and incorporate the most relevant cultural features 

into the evaluation design. » 



. SETTING UP EVALUATION DESIGNS 

In this section we yill highlight sere of the key methods and 
basic considerations in building in evaluation t<\jjxevention 
program designs. Many of • the steps' illustrated were discussed 
in greater detail earlier. The purpose here is to assist you in 
placing all *of the information presented in its appropriate ^ 
perspective. 

Steps in Program Evaluation , 

Exhibit 5 gives an overview of eight basic "steps in program 
evaluation which include: 

i * « 

• Program* Readiness ° 
^_S^ Needs Assessment 

• Setting Goals f 

• Measures 

• Designs 

• Data Analysis*, 

• Reporting 

• Use ♦ 




\ 



The subsequent illustrations and discussions include addi- 
tional details about 'each step. This will provide program planners 
^with an M at-a -glance" description of these vital elements of an 
overall evaluation design. 

^, 

The illustrations which follow were adapted from: Mitchell, 
M.E., 'and Rottner, L.H. Program Evaluation Handbook for Substance 
Abuse Professionals . University Park: Addictions Prevention 
laboratory, Pennsylvania, State University, College of Education. 
February 1979. Artwork by Noel vaitder Veen and Kay Anderson. 
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v Organizational Readiness ' * e 

, Exhibit 6 depicts the A VICTORY Model , developed for the 
National Institute of Mental Health, to determine the readiness of 
the program for an evaluation. As presented in Exhibit 5, program 
readiness is a result of defining the use of the evaluation, describ- 
,ing reporting requirements of the evaluation data defining the data 
analysis plan, identifying the appropriate measures, setting goals 
and objectives, and assessing needs. The A VICTORY model further 
defines program readiness through a series of questions. Each of 
theie questions should be thoroughly considered in the development 
phase qf the evaluation design. Reflecting on each of these 
components can also uncover possible traps or pitfalls which could 
occur. "Adding a positive note to a sometimes time consigning and 
complicated process, the A VICTOR Y model suggests that the program _ 
can be victorious in' con^leting~Infomation and useful needs assess- 
ment and evaluation- studies; 



A - Refers to the organization's ability to conduct evaluation, 



V - fchat things are Valued lyy the program and the culture it serves? 

What information is currently available about the program? , 
What additionaT^research has/to be' conducted? 

C - What circumstances , political and other will surround the 
operation of* the program being evaluated? 

T - Timing ; where is the program in terms of its cycles? i.e.,- 
the proposal writing stage* program start-up, final reporting, 

0 • Whose obligations r are being^net? 4*ie program director, the 
funding source, the general public? # 

R - Resistance ; what are inhibiting forces? Are staff intimidate^? « 

Y - Yield; what are the expectations of evaluation to be conducted? 

4 

f t * 



( 
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A - ABILITY 



V - VALUES * 
( I - INFORMATION 

£ - CFRCUMSTANCES 

, T - TIMING 

T 0 —OBLIGATION 
1 

R - RESISTANCE 



Y " YIELD 




Needs Assessment , * 

This ijnportant step has been covered, adequately in a separate 
7°° kl f? } 1 ^ lls series ^ in other sections of .this booklet. 
(See BAiblt 2). At this stage in planning, however, a review 
of nee<fe assessment data may Help the evaluator to focus on * 
essential problem areas. Program staff may be able to provide 
additional information whidh will also improve upon attitudes 
about the readiness of the progkm to do the evaluation. ^ 

Goals T^|d Objectives 

This step has also beeincovered in other sections (see 
Exhibits 4 and ?). This, HBWever, is another step in planning, 
whereby a review which involves program staff, can be essential 
to setting up an effective design, and improyijig*jup0n_Jthe 
readiness of the organization. 



■3 4 
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EXHIBIT 7 - / 
ami s Ann objectives 



1. DECIDING WHO SHOULD BE INVOLVED 



SPECIFYING MEASURABLE 
\ OBJECTIVES 



2. GENERATING PROBLEM STATEMENTS 



" 3. IDENTIFYING PROGRAM 
. GOALS 



/ 




5. ASSIGNING PRIORITIES 
FOR EVALUATION 



Measurement in Evaluation 



A blending of the assets of quantitative and qualitative 
measures may be the most viable strategy for evaluating ftulti- 
cultural drug abuse prevention! programs. It appears to hold the 
most promise for being responsive to both the funding agency and. 
the minority ccrranunity. 

NIBA's Prevention Planning Workbook states .that it is 
Vdesirable* to measure a complex social phenomenon like drug abuse 
in several ways, that findings are more certain if the data are 
confirmed through several different methods such as questionnaires 
direct observation, and analysis records (French^t* al., 1979). 

There are several other reasons for this multiple -measures 
approach. First, most programs with a prevention orientation 
typically cannot expect massive changes in -client behavior or 
attitude. It is more likely jthat some particular segment of a 
client f s attitude or. behavior will be altered by the program 
expgjTence; many others will not. Similarly, change along certain 
criterion dimensions will vary according to the individual. One 
client may improve school performance through better grades, 
another through increased participation in extra-curricular 
activities. ' * 

^iilt iple measures also provide for the opportunity to detect 
latent changes in clients. Changes which have not yet been * 
manifested in behayior may be detected at the attitudinal -value 
level. This advantage is especially important in trying to 
evaluate program effectiveness over a brief period of tijne. For 
example, when av6npting to evaluate the impact pf a values 
clarification session, one often cannot afford to wait a] period 
of t several months or years to determine the full behavioral 
impact of the sessions. Rather, it -is desirable to determine if 
the client has begun to reassess, if only mentally, his or her 
system of values. Instrument-based measures can be useful for 
such 'purposes. 

Drug abuse prevention progran evaluations have successfully 
utilized such approaches afs direct. observation, structured 
interviews and diaries kept by participants in conduction with 
stjucturea^questionnaires. These methods allowed the e^aluatdrs 
,to provide a very convincing picture of program impact as well as 
/valuable an<} reliable recommendations. 
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1, BEHAVIORAL OBSERVATIONS 

2. PROGRAM RECORDS 

,3, WRITTEN "INSTRUMENTS 
ACHIEVEMENT TESTS 
APTITUDE TESTS 
INTEREST INVENTORIES ' 
ATTITUDE SCALES 
QUESTIONNAIRES 
INTERVIEWS 1 

PUBLIC DOMAIN RECORDS 



4. 
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Selecting Instruments 



•Exhibit 9 highlights the key points to be considered in 
selecting appropriate instruments for measuring program process * * 
and outcomes. A consideration among many, frcgLi^cultural perspec- 
tive is the intended use of the data collected^TEifority conmuni- 
ties respond best when they are given assurances that they can have 
access to results of any questionnaires given and that such 
information will not be used in anyway to their detriment. 

There are several sources for measurement instmnents which 
may be useful for a program evaluation. NIEA 1 s Prevention 
Evaluation Guidelines discusses considerations in the selection 
of measurement instruments and gives seme examples. NIDA's Drug 
Abuse Instrument Handbook lists 2,000 items from forty instilments 
•and categorizes tftem in tne following areas: 

• demographic variables; 

• interpersonal variables; 

• intrapersonal variables; and 

• drug variables. 



Miller* s (1978) Handbook of Research Design and Social « 
Measurement presents a general discussion of social researcn and 
a more detailed discussion of socicmetric scales and indexes frith 



examples of instrunents. The discussion of the instruments is 
divided into the following categories:, 

p • social status;' 

• group structure and dynamics; 

• social indicators; 

/ 

+ measures of? organizational structure; 

/ « ! • / „ . ♦ 

• evaluation research and organizational effectivess; 

• conmunity; 

• social partiGipation; 
m leadership ih the work organization; 
' • morale and job^satis£action; 

scalfes of attitudes, values, and norms; * 

• family and marriages; 
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personality measurements; and 

inventories of socicmetric and attitude scales. 




Constructing .Measures * j ** 

'Sane traditional measures can often^ be* modified and indeed 
sane new oites can be developed tq meet the program evaluation 
needs. Certainly technical assistance should be sought for more 
comprehensive* measurement devices. / 

y ^ Sometimes, however, program staff may need to develop intake 
^questionnaires, survey sheets or other simple measurement devices. 



B&iDit 10 illustrates several suggestions in .constructing such 



measures . 

• Parsimony - Use as few words as possible. 

• Specificity*"- Get to the point. 

• Singularity.- Let each question or item speak to one 
' . ' V subject only. 

* • * 

• ' Simplicity - Avoid the complicated or abstract. 

• Sensitivity - Be careful not to offend, culturally or 

otherwise. ' , 

* * ' ** ; 

• fc « W Semantics * Make sure that the language is standard for 
the population served. 



• Slant*- Retain as much objectivity as possible. 



m 
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EXHIBIT 10 . • 
* ffflSTmiCTINfi. MEASURES" 
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Designs 0 * 

Experimental designs are, most rigorous in controlling for 
rival causes and represent the ideal -desi^i. This design uses at 
least? two groups, /one that takes part in the project (i\e., 
experimental group) and one that does not (control group) . If the 
experimental grcyp performs better than the control group (gets 
better grddes, has fewer disciplinary problems, holds jobs longer and 
if participants/ were randomly assigned to the two different groups, 

. we could be fafrly certain that the project "caused" the improved 

performance. The b,ig ingredients in experimental designs is the 
\ ran dom assignment of clients to the experimental and control 

Xg^Jups* (i.e., clients- have^an equal chance of being assigned to 
either group), because randomization controls all other rival 
causes. While randomized experimental designs are rigourous and 
scientifically elegant, unfortunately they often are impossible to * 
execute in action settings because of the resources required, 
impractically, and ethical considerations. Exhibit 11 illustrates 
the various formats for #iis type -of evaluation design. - 

Quasi -experimental designs do not satisfy the strict required - 
ments ot scientific experimentation, largely* because of the 
inability to randomly assign ^eonle^to projects. Consequently, they 
do not control for 7 all outs^ie effects as effectively* as randomized 
experimental designs, usually leaving one or several rivSl causes 
uncontrolled. Quasi -experiments do have the a&tfantage of being 
practical when conditions prevent 'true experimentation. Some • 
quasi-experimental designs use available individuals or existing 
intact groups with similar characteristics to project clients in 
iieu of randomly formed groups. Nonequivalent control groups are 
COTmonly used in evaluations and a- major issue in their use is how . 
✓to make the comparison group as similar to the experimental as 
, possible. Another often ifeed quasi-ex^rimental design is a time 
m series where a series' of measurements for an experimental project 
. and a similar group is taken at periodic intervals before and after 
the interventions. This approach identifies whether there has been 
any change in patterns. - ~ 

Pre -experimental designs are available when it is impossible "to " 
use even quasi-experimental designs. Their principal characteristic 
involves comparing a pre-prtfject against a post-projed: outcome 
measure.*" Such a design provides insight into how effectively the 
project is delivering services, especially during the early stages 
of the project or when there is intense pressure to produce some 
kind of evaluative data within a short period of time. The inherent 
weakness of pre -experimental designs is that they fail to control 
for many rival causes. Thus, they leave considerable room for 
differing interpretations of how much change has occurred and how , 
much was due to the operation of the project s 
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. EXHIBIT 11 
•EXPFRIffiWnU.IFSTCTK 



• Prp Tfst-Pqsttfst - Cctjtpol Group Design ; 

Random assignment Veasure Program Measure 

Measure - - Alternate Measure 

Program 

• Pqsttest Only — Control Group Design 

1 Random Assingment Program ftASURE 

% ■■ - Alternate treasure 

Program 

V 



• ' - 'EXHIBIT if' 
. QUASI- - EXPERIMENTAL DESIGNS 

m . TIME SERIES DESIGN 1 •** 

MEASURE 1 — > 2 — > 3 — > PROGRAM — > MEASURE 4—^5 

f. tiqN EQU I VALENT CONTROL GROUP PES I6N 

MEASURE - - > PROGRAM - - * MEASURE * t/A . 
'1 MEASURE ALTERNATE PROGRAM - ^ MEX^R$4/ l % 



EXHIBIT 13 ' 
P-RF-FXPFRIMFNTAt. RESIGNS 



• ONE GROUP PRE/POSTTEST DESIGN 

MEASURE PRbGRAM —> MEASURE 



t STATIC TGROUP. COMPARISON ' 

PROGRAM • — MEASURE 
» > _ MEASURE 





t ONE SHOT' CASE STUDY 



PROGRAM 



-> MEASURE 
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Data Analysis ' ' f " 

Preliminary data analysis may>be dgge intermittently 
throughout the data col'lectiotf period are 'monitoring strategy. 
This is especially useful when qualitative and quantitative 
methods are being combined beca u se each approach can be used to 
validate the findings of the other as the data become a^ailablS. 

Myers (1977) states that one of the deficiencies 
conventional procedures is that data analysis and interpretation is 
done at the exclusion of the interviewers and other people who 
have first hand knowledge of the respondents. He suggests that the 
analysis should evolve #l frcra shared understanding of social -reality 
anting researchers, consultants, interviewers, and respondents. 
AsSuijptions that- each exchange occurred under identical circumstances 
are not i^sed f L r l(Myers, 1977, p,247). Myers used indigenous 
interviewers who provided input throughout his research study. 

A very comprehensive, useful evaluation Report can be produced" 
by combining these qualitative and quantitative findings and 
focusing on the research questions in the Analysis of the data. 

In addition to tke above important theoretical considerations^ 1 
some program planners may find the need to make some sinple \y 
analyses of data collected. Exhibit 14 lists some simple descrip- 
tive techniques that can be used: . « 



1 . Dispersions 

/ 

2. Percentages 

3. Averages 
4/ Frequencies 



The spread of responses usually indicated by 
quantile deviation, standard deviations, 
range of. 90-10 percentiles, etc. 

The proportion of. times in any given set of 
scores that any one response occurs. 

The stm set of responses or scores divided 
byjthe total number of scores. 

A tabulation of scores from high to low 
(or low to high) showing the number of 
persons who obtain each score or group of 
scores . 
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Presenting Data and Reporting Evaluation Result s 

The results of the evaluation (both process 'and outcome) fc 
should provide information on 1) whether or not the. program 
succeeded inking what it was planned to do, 2) how well it did 
it, and 3) new information discovered* TTie results may not 
identify a clear path to program inprovement , but they should 
provide information on the program's strengths and weaknesses ^ 
which can suggest strategies for improving effectivenessr 

"The importance of reporting the evaluation results cannot be 
overstated since it has the functions of: 

• being the official tecord of the evaluation; 

6- * 

• answering the evaluation questions; 

• describing the program being evaluated; 

• describing thfe steps in the evaluation; 

• explainingvthe^ procedures used; j 

• presenting the evaluation findings; / , 

• drawing conclusions about the findings; 

• identifying a program's strengths/ 
pointing out a progran^s weaknesses; 

identifying »*»»Te ch ange is needed; 




• establishing ^a basis for making-crucial decisions; and 

• presenting new findings* ^ 

Varioiis audiences for the report should be considered when _ . 
writing it* Audiences may incKxie federal, state, or local 
administrators, the program staff and participants, and members of 
the ccnramity. The following separate volumes, of the report should 
be considered: x 

« 

Part 1. An overall^ Executive Sunrn3ry for administrators and 
decision-makers* The report should not exceed ten pages. It 
should be easily readable and free of jaTgon. The evaluation _ 
results and possible recommendations should be presented within 
the context of national and state prevention policy. * 

Part 2* A full Technical Report for administrative technical * 
staffs , evaluators of other programs, and future evaluators of 
this program* The report should include all of the procedures used 
throughout the evaluation process in detail. It should also include 
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INITIAL PLAN 



EXHIBIT 16 
RFPORTINR EVALUATION RESULTS 



PURPOSE 



QUESTIONS 



AUDIENCE 



EVALUATOR ROLE 



RESOURCES 



FINAL PLAN 
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the findings, suggestions for program improvement, and recommenda- 
tions for future evaluations of thfe program, ^ 

-Part 3, A report to th^^mgmity. This'j-eport should be >non- 
f technical , It should be presented in a style and language that 
/ is comprehensive and interesting^© the ethnic coranmit^-- Often 
a format that presents sunmary findings interspersed with short 
case studies is the most effective. "See Exhibits 15 and 16. 
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Hie purpose of this booklet is fc to encourage the thoughtful 
use of evaluation as a* management tool. Three levels of evaluation 
are discussed: process, outcome, and impact. A process evaluation 
is used to assess program implementation. An outcome evaluation 
assesses the short term or immediate effects of. the program on the 
target, population while overall program effectiveness is the concern 
of an % imp ac £ evaluation. These levels of evaluation axe not 
necessarily mutually exclusive. Impact evaluation^ however, require 
long-term, sophisticated methodologies which are beyond the ' 
resources of most individual programs and are, therefore, not 
^ discussed at any length. % 

Regardless of "the level of evaluation* it is essential that 
the planning and implementation consider the cultural and socio- 
political context of the program. Equally important, the evalua- 
tor, whether thitfd-party or internal, must be sensitive to that 
context. S/he must work closely with the program's staff in 
reviewing the objectives to ensure that they are feasible, 

i cultural 1/ relevant, and measurable. It is critical to the 

success of the evaluation effort that the evaluator and program * 

1 staff agree upon what is to be measured and the criteria for 
success. . 

• ■ * 
Once the criteria for success are clearly defined, the 
program staff and evaluator can determine what data are needed and 
how they will be obtained. Some data may already be available 
frcm the needs assessment and other sources in the community. 
Additional data may be obtained through utilization of multiple 
u ^pLiantitative and qualitative approaches. Since many standard 
instruments are not relevant *or^aHd for various ethnic groups, 
it may be necessary to revise^hem or\everi develop new ones. 
Various technical assistance resources>are available to aid 
prevention programs in doing so. 

An appropriate design fo/ the evtluftion must also be selected. 
NIDA's Prevention Planning Woi^qk^ gfcails three: experimental, 
quasi -experimental, and pre- experimental . Experimental designs 
are" the mo£t scientifically rigorous but^Of ten cannot ba implement- 
ed in the real world of social change^ programs. Quasi -experimental 
designs are more flexible and usually nwnj/ feasible for prevention 
programs. , yf* 

^ Data collected in the evaluation must then be analyzed and 
prepared for presentation. 

Basically, there, are two approaches to analyzing datai 
- quantitative and qualitative* The quantitative approach is the 
analysis of numerical dat^f Qualitative data which involve 
descriptive, le^mea^rfable subjective matters should not be 



overlooked.'* Regardless of the specific techniques, data analysis 
fust make allowances for controlling culturally sensitive variables, 
' Although analysis techniques may be«purely statistical and culture 
free, interpretation of the data is not. 




_ jt ion findings can tjjen be presented in -a series of 
Feports^RJsigried for specific audiences, Ari executive summary 
might be prepared for policy and d^sion-makers; a detailed . 
technical report for program staff and other evaluators; and a 
third report for the ethnic community. 

"* * • 
Well designed, culturally relevant evaluations of drug abuse 
prevention j^ogjams lead not- only to program improvements but also 
to advancements ifpthe state of the art fc in^revention and ^valuation 
research in multicultural 'connnunities , 

£ 
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APPENDIX A 
MULTICULTURAL SURVIVAL INVENTORY ^ 



I. PERSONAL - f 

A. AGE 5 



B. SEX 

C. AGENCY 



D. IDENTITY (HOW DO YOU PERCEIVE YOURSELF) 
II. CULTURE (OWN/WORKING WITH) 



_ A. VALUE SYSTEM (WHAT'S IMPORTANT) 
• < l. : ^ 

3. 

H.^l 

5. 



*B. MAJOR PROBLEMS/CONCERNS' 

9 ' . 



1. 

2. ( 
3. 
4, 
5. 



^- C. POSITIVE ATTRI 

« 1. 

2. 

• 3. _J 

4. 

5. 




DEVELOPED BY ItRON KUNISAWA, MTO ASSOCIATES, SAN FRANCISCO, CALIF. 
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D. NEGATIVE ATTRIBUTES 

l._ : 

2 : . ! 

3 , . 

4. ' : 

_5. s £ 

E. MAJOR DIFFERENCES FROM THE DOMINANT CULTURE 

1.' 



2. L 

3. _ 

4. _ 

•5. _ 



F.' IDENTIFY FACTORS OF SUPPORT FOR AN INDIVIDUAL 

1; 

2 . 

* 3 : 

0 

1 

• • 5; L_ 



6. IDENTIFY FACTORS THAT CONSTRAIN OR HINDER INDIVIDUAL GROWTH 
AND DEVELOPMENT (iF ANY) 



- . 2, , : 

3. t : « 

t\ __ : 

5 : . 
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H. -IDENTIFY THE FIVE MOST -STRESSFUL ELEMENTS THAT EXIST FOR - 

PEOPLE IN THE CULTURE » h 

- * •'• !. 



X 



• 5. 



I. IDENTIFY FIVE CULTURALLY ACCEPTABLE WAYS FOR COPING' WITH THE 
ABOVE STRESSFUL ELEMENTS 

i. . ^ ; 

* 2. - ' 

3. : ' - _. ' 

4. : e : 

5 ^ 

J. IDENTIFY WHAT IS HIGHLY PRIZED AND/OR RESPECTED BY THE CULT<JRE 

1. ^ 

2. : 

3. - 

. 4. | . 

5. ' 

K. LIST THE FIVE MOST DIFFICULT ASPECTS ABOUT BEING A FEMALE IN 

THIS CULTURE ■ ■ 
" 1/ : . 

2. ! ! ^ 

3. . < . - 

ft. / 

" 5 
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L. LIST- THE FIVE MOST DIFFICULT ASPECTS ABOUT BEING A MALE 
IN THIS CULTURE 
• 1.^ — , — 

2 : - 

' 3. 

t ! 

; 5 



III, SURVIVAL IN THE DOMINANT CULTURE (WESTERN EUROP 

A. SOCIAL PROBLEMS (INTERACTING, RELATING TO,\etC.) 
1. . '. 



ERLC 




' 2.1 
3. _ 

i _ 

5. _ 



J. 



B. ECONOMIC PROBLEMS 
1. 



? L 

L ' r 

3. " 



5. 



-\ 

C. POLITICAL PROBLEMS 
. 1 

' 2. _^ 

3. 

4. : 

5. _^ 
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D. FAMILY RELATED PROBLEMS-' 
1. 



2. : e t 

3. 

4 

5. 





e. professional/career problems 

1. 

2. • 

3. 

4. ■ : 



Ft IDENTIFY THE FIVE MOST IMPORTANT "SURVIVAL SKILLS (iN RANK 

~ V 51 ' * .SS - i - -i • or » s -O! 

order) . c 7 , 

2. ' * 

- 3. : iiL ! 

4, f , ^ 

5. ' 
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v * ' APPENDIX B , 

" ' GANTT CHART OF PRELIMINARY EVALUATION ACTIVITIES 

This GAhHT chart illustrates a time frame for the 'first 
evaluation of a relatively large program. It can be adjusted for 
snaller programs. This chart should be integrated into the overall 
program management GANTT chart. It is based on the assumption that 
a pce/posttest, or time series approach -and at least one comparison 
group will be used. It is also assumed that a third-party eval- 
uator will be used. These assumptions are undoubtedly idealistic 
for most programs, but a walkthrough the GA-VIT chart will illus- 
trate some of the tasks and issigs that ndst be addressed in the 
first year of an evaluation. 

In Task 1 the evaluator 'is hired. If a third-party evaluator 
is being selected through competitive bidding, qt least one month 
will be required to accept and review bids. Thtee weeks are 
allotted for refining the evaluation plan (Task! 2). This is based 
on the assumption that a preliminary, relatively^ general evalua- 
tion plan was developed and incbrporated into the management 
plan prior to program start-up. Such plans ?re Often required in 
a proposal. However, the evaluator working w||h $he project 
director should refine the plan and develop it^ui Sore detail. 
This should include formulating a series of research questions 
t&t -are to be add*ocs2d. . The research questions should ,fee t , 
geared to the informational needs of decision-makers and other 
aftiiences. / 

Questionnaires, interview guides, and other evaluation 
instruments wimtake some time to develop of obtain. The GANTT 
chatt allows five weeks for this task' (Task'3). Concurrently, 
pi&ns should be made for managing the evaluation (Task 4) . For 
Example, a recordkeeping system for the evaluation should be 
designed so that it is compatible with the overall recordkeeping 
system of the program. The evaluation* should be planned so 
that much of the data are routinely collected and recorded in 
order to minimise the data collection burden on the staff. This 
holds true particularly for process data such as times and typqs> 
of services provided to individuals or cooperative arrangements 
made with other agencies. The Collection of superfluous data 
should be avoided because it can become too cumbersome. 

- The GANTT chart provides two weeks for the task of identify- 
ing criteria for the selection of the pilot test comparison group 
(Task 5). The 'selection of participant and comparison groups 
for the, pilot test (Task 6) may take a month. Staff should then 
be trained for the data collection ffask 7). Three months are 
provided' in the GANTT chart for the pilot test data collection 
(Task 8>. This time line is" based on using one participant 
group and one comparison with thirty or more persons in each 
group* Two months -are scheduled for data processing and analysis 
of the results (Task 9) . 
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GANTT CHAttT OP PRELIMINARY EVACUATION ACTIVITIES 



EVALUATION ACTIVITIES 



REFINE EVALUATION PLAN 



Task 



1. 
2.' 
3. 
4. 
5. 



ftire third-party evaluator \ 
Refine plan \ 
Develop/obtain evaluation instruments \ 
Develop evaluation management system V 



Identify criteria for selection of pilot 
test comparison group . * 

PILOT TEST EVALUATION STRATEGY 



r* 6. Select participants and comparison groups 

7. Train staff in data collection 

8„ Pilo^test 

9. Analyze results 

PREPARE FOR EVALUATION 



10. 
11. 
12. 

13. 
14. 



15. 



Revise/finalize evaluation plan 

Select participant group 

Determine salient characteristics of 

comparison group 
Select comparison group 
Order/revise/develop final evaluation 

instruments 
Develop evaluation schedule 



PREPARE 1ST YEAR EVALUATION REPORT 



MONTHS ' 



10 



11 



12 



Plans/reports 




\ In Task 10, approximately six weeks are provided to revise 
and Vinalize tfte evaluation plan for the next year. The selection 
of participants for the evaluation (Task 11) can begin prior to 
finalizing 'the second year's evaluation plan* Criteria for the 
selection of participant and comparison groups should be included 
.in the evaluation plan for the second year. The most salient , 
characteristics for matching program participants to comparison 
w group meimbers (Task 12) can be determined in the analysis of the 
* pilot teat results. When the pilot test is conducted* participants- 
and comparisons are usually matched on what are assumed to be tl\e 
most ijnportant characteristics. Following the pilot test, it 
should be^possible to select these groups on the roost relevant 
characteristics. Comparison group members can be selected (Task 13) 
following completion of the previous task. , 

Evaluation instruments are ordered, revised, or developed in 
the eleventh month (Task 14), and the schedule for the next year's 
evaluation is developed in the last six weeks of the first year in 
Task 15. The first year evaluation report will consist of the 
results from the pilot test and plans for the next year's evalua- 
tion. 

Some** of the most important things to remember when developing 
evaluation are: 

• to use the first year of the program for evaluation 
planning and pilot testing; 

• to use a comparison group or groups if possible; and 

• ijot to expect credible evidence of effects on program 
participants the first? year of the eyaluation. 
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